ITF in Science mode for all the shift.

Guard tours (time in UTC)

From 21:00 to 21:30; from 23:00 to 23:30; from 1:30 to 2:00; from 3:30 to 4:00

ITF in Science mode for all the shift

Software

At 22:27 UTC I stopped and restarted the process ELE_QGBT because it was not running porperly.

Guard tours (time in UTC)

from 21:50 to 23:20; from 00:00 to 00:30; from 2:00 to 2:30; from 3:50 to 4:20

ITF found in Science.

At 15:14 UTC the DMS reported a jump in the power DAS OUT; I immediately contacted the TCS on-call. From remote Ilaria adjusted the TCS WI chiller setpoint and thanks to this operation we could recover the standard value of the power DAS OUT without breaking Science mode (see entry 64172.

At 20:34 UTC ITF in Calibration for the planned calibrations (the planned starting time was delayed by LIGO):

• CALIBRATED_DF_DAILY, completed.
• ISC injections: inject_asc_test.py, completed.

At 21:05 UTC ITF back in Science mode.

No DMS alerts in the shift.

Francesco called me because of the change of state of WI CO2 laser.

To restore the standard condition, I adjusted the WI chiller set point from 19.04 degrees to 19.05 degrees (15.40 UTC).

The BCK chiller set point has been chaged accordingly.

ITF found in Science mode.

At 19:15 UTC the Squeezing was disengaged and the ITF exited from Science mode. Unfortunately, the ITF switched out of Science mode while I was at dinner; when I returned, I checked the cameras and the main screens of the control room and the ITF resulted locked and in standard condition thus I assumed that it was still in Science mode; I realized after a few minutes that we weren't in Science mode.

At 20:02 UTC I re-engaged the Squeezing and then I set Science mode 20:02 UTC.

Software

• FCEM_MotSwitch restarted at 16:52 UTC, at 17:02 UTC, at 17:16 UTC after crashes.
• at around 17:09 UTC I restarted SuspEBMoni to update the cfg file to remove the grey flag SDB2_SBE - rms(SBE_SDB2_SA_F0_diff_LVDT_z,5)

Guard tours (time in UTC)

From 17:15 to 17:45; from 20:05 to 20:35.

ITF found in Science mode.

At 8:44 UTC the DMS started to report a red flag for WI_CO2_Laser - TCS_WI_CO2_PWROUT (see entry 64156).

I contacted the TCS on-call and from remote she adjusted the WI chiller set point, then at 9:17 UTC, we set Adjusting mode and she restored the standar power; ITF back in Science mode at 9:20 UTC.

ITF still in Science at the end of the shift.

ENV

Sea and windactivity increasing.

Software

At 12:50 UTC the VPM instance became frozen; it was restarted at 12:52 UTC.

ITF found in relocking; it relocked at 5:35 UTC and it remained in Science mode for the rest of the shift.

Guard tours (time in UTC)

From 6:00 to 6:30; from 8:00 to 8:30; from 10:00 to 10:30; from 12:00 to 12:30.

The earthquake is the following: M 4.4 - 4 km W of Rakovica, Croatia.

ITF found in Science mode.

Sstarting from 22:40 UTC a series of local earthquakes unlocked many times the ITF:

• Unlock from Science mode at 22:40 UTC  ( ML 3.1 del 27-04-2024 ore 00:39:58 (Italia) in zona: 3 km NE Barberino di Mugello ).
• In the next relock another one ( ML 3.1 del 27-04-2024 ore 00:58:46 (Italia) in zona: 2 km NE Barberino di Mugello ) unlocked the ITF in CARM_NULL_1F.
• The ITF was back in Science mode at 23:44 UTC but antoher one ( ML 3.0 del 27-04-2024 ore 01:56:23 (Italia) in zona: 2 km NE Barberino di Mugello (FI) ) unlocked again the ITF at 23:58 UTC.
• The ITF was back in Science mode at 00:49 UTC bu another one ( ML 2.8 del 27-04-2024 ore 02:52:48 (Italia) in zona: 2 km NE Barberino di Mugello (FI) ) unlocked again the ITF at 00:53 UTC.

The ITF was back in Science mode at 1:42 UTC, still in Science at the end of the shift.

Guard tours (time in UTC)

from 21:50 to 22:20; from0:00 to 0:30; from 2:00 to 2:30; from 4:00 to 4:30

ITF found in Science mode.

The ITF unlocked at 2:22 UTC and the unlock seems to be correlated to a problem of DAQ/PR; see attached plot and the log file of ISC_Fb:

2024-04-26-02h22m33-UTC>ERROR..-[TfbSource::StoreData] producer Sc_PR: overflow at GPS 1398133371.999912050 (received 2021 / expected 2000) (1323 times)

This kind of behaviour is something similar to what already reported in the entry: 63958.

At the next relock the lock of the CITF took more than 30 minutes (I also performed the manual prealignment twice); once the CITF was locked LN3_SQZ was achieved at the first attempt and Science mode was set 3:58 UTC.

Gurad tours (time in UTC)

From 21:30 to 22:00; from 23:30 to 0:00; from 1:30 to 2:00; from 3:15 to 3:34.

ITF found in Science mode, LN3.

At 15:30 UTC ITF in Calibration mode for the planned daily calibration.

At 15:43 UTC ITF in Commissioning for the SQZ_INJ vs CC phase scan activity; after this activity the ITF has been locked in LN3_SQZ.

At 19:23 UTC ITF in Science mode.

Software

FbmMainusers restared at 17:05 after a crash.

VirgoOnline vpm instance became unreachable at 14:52 UTC; restarted by Emmanuel.

HVAC

new transformer (220V-->24V DC) of the DET AHU installed today, see 64093

Guard tours (time in UTC)

from 17:50 to 18:20; from 20:20 to 20:50.

SHIFT GOAL:

The goal was to perform several CC scan at different SR angle. Nevertheless, due the not optimum condition of the MZ and OPA (waiting for the retuning) we decid to perform a couple of scan without changing the SR angle to check the SQZ alignment-

SHIFT activity:

We started at  16:00 UTC and to not interfere with the ITF and risk to unlock, we manually injected the SQZ. 

• System initial status: ITF LOW NOISE 3; SQZ_MAIN @SQZ_Locked_NO_FC

• 16:10:39 EQB1  fast shutter opened (process: SQZ_ctrl/shutter ‘open’)

After that the CC_loop was engaged and then  the AA (process: SQZ_PLL_AA/ Matrix and Filter ‘AA dither enable’).

• started a CC 4MHz phase scan @ 16:52. 

  • time per point = 6 s

  • - initial phase  = 20 degree

  • - final phase = 230 degree

  • - phase steps = 1 degree

  • - random phase = False

  • - Closed shutter sleep time = 120 s

  • - AA closed on dither = False

  • - cc loop gain   = 75000.0

N.B: No glitch during the scan, it was just before and after (see plot)

In meantime Henning checked the MZ and changed lowered again the offset at 0.035 V due to the humidity  misalignment that spoiled the contrast and induced the set point near the actuator saturation. After the Henning action we redid the scan. 

We stopped at 19.25 after that some data have been collected at the CC phase value corresponding to the SQZ and ASQZ, or better to the minimum and maximum values in the BRMS range @210 MHz and in other ranges.

Further data  and plot in the following comments.

ITF found locked in LN3; the Squeezing was not working for temperature transient started in the morning due to the planned work at the UTA.

ITF was in Science but the GWISTAT page was displaying VIRGO in down; at the daily meeting it was discovered that it was due to a bug introduced by Hrec connected on FbmFE.

At 13:45 UTC Hrec process was restarted by CAL team with the previous version; this fixed the issue.

At 14:00 UTC ITF in Calibration to perform the daily calibration.

At 14:14 UTC ITF in Commissioning mode for New filters for marionette reallocation; activity concluded at 18:39 UTC.

At 18:39 UTC ITF in Science mode in LN3; because of the thermal transient in in progress in DET LAB the Squeezing could not be engaged.

At 20:30 UTC the ITF unlocked; relock in progress.

DET

At 16:10 UTC DET_Main went in unknown state after an unlock; the ITF was put in NI single bounce for safety and then the situation recovered with the procedure provided by Romain.

HVAC

The work at the HVAC DET was completed around 17:30 UTC; temperature and humidity inside the standard values.

QNR

Martina tried to recover the Squeezing but she was not able to do it; investigation is in progress.

Software

At 16:48 UTC BACnetServer restarted because it was not providing data.

6:00 UTC: ITF found in BAD_WEATHER due to high wind and sea activity.

At 10:13 UTC I removed the mode BAD_WEATHER because the related flags WIND and SEA on the DMS were not red anymore but unfurtunately the lock acquisition continued to fail at various state; Maddalena checked from the control room.

After a check we discovered a change in the WI CH power that happend last night thus, at 10:30 UTC, I set TROUBLESHOOTING and we contacted Ilaria to restore the good value of the powers. This problem was not reported on the DMS thus a readjustment of the thresholds is needed.

After this change the interferometer continued to unlock also due to a further worsening of the weather conditions: strong storm with lightning and thunder.

The commissioning team present in control room is investigating at the problems.

ALS

As reported in the previous shift the DMS is reporting a problem with WE ALS Laser; this morning I informed the experts by mail. At 12:00 UTC I did not receive any answer thus we decided to contact the ALS on-call; unfurtunately he is not available to look at the problem.

SUSP

At around 12:50 UTC WE and WI LC were opened by the guardian after an unlock. I tried to close the loops but the new GUI interface did not work; for this reason I contacted the SUSP on-call. Investigation on the problem in progress from control room.

ITF in Science mode for all the shift.

Guard tours (time in UTC)

From 6:00 to 6:30; from 8:00 to 8:30; from 10:00 to 10:30; from 12:00 to 12:30.

ITF found in Commissioning mode for: Change of logic for the LOW_NOISE_1 and LOW_NOISE_2 to bypass LowNoise1 actuation stage; activity concluded at 21:00 UTC

In parallel, Romain and Victor were investigating the excess noise on B1_PD3 that was preventing to lock the OMC thus at 21:15 I set Troubleshooting.

At 23:14 UTC we could lock the OMC but we unlocked during the transition to LN2; unfortunately the excess noise came back after the previous unlock.

Diego worked up to 1:00 UTC trying to understand the problem but we kept unlocking during the lock of the OMC; for the next lock acquisition we decided to wait more time in CARM_NULL_1F and try to perform the trick of the SR but unfortunately the ITF unlocked again.

The lock of the OMC kept failing for the next two hours; then at, around 3:00 UTC, the OMC locked again but the ITF unlocked LN2.

At the next attempt the ITF reachedn LN3_SQZ thus at 4:01 UTC I could set Science mode.

Guard tous (time in UTC)

21:00-21:30; 23:00 -23:30; 1:00-1:30; 3:!0-3:40

DET
(16-04-2024 19:30 - 23:15) From remote
Status: On-going
Description: unlock at DC_READOUT step

ITF found in Science mode; it unlocked at 23:28 UTC because of the following earthquake: M 5.9 - 143 km SE of Koshima, Japan.

The ITF was relocked at the second attempt; Science mode at 00:39 UTC.

At around 3:00 UTC the DMS started to report a red blinking flag for Sc_WE_MIR_COIL_FLAG (plot2).

At 4:16 UTC the ITF unlocked; the unlock seems correlated to Sc_WE_MIR_COIL_FLAG (plot3); to be checked by experts.

ITF left in relocking; no DMS alerts.

ENV

Sea activity increased during the shift (plot4).

Guard tours (time in UTC)

From 21:00 to 21:30; from 23:00 to 23:30; from 1:00 to 1:30; from 3:12 to 3:44.

ITF locked for all the shift.

From 18:30 UTC to 18:40 UTC ITF in Calibration mode to perform the daily calibration.

No DMS alerts.

Software

FCEM_MotSwitch reatrted at 17:43 UTC after a crash.

Guard tours (time in UTC)

From 13:00 to 13:30; from 15:00 to 15:30; from 17:40 to 18:10; from 19:40 to 20:10.

ITF found in Science mode.

From 18:30 UTC to 18:40 UTC ITF in Calibration to perform the planned DAILY_CALIBRATION.

At 19:35 UTC the ITF unlocked;  in the attached plot you can see a spike in the Sc_PR signals. At the same time the process ISC_Fb reported the following message:

2024-04-13-19h35m31-UTC>ERROR..-[TfbSource::StoreData] producer Sc_PR: overflow at GPS 1397072149.999912000 (received 2230 / expected 2000) (14490 times)

The ITF relocked at the first attempt; Science mode 20:27 UTC.

SBE

With the ITF unlocked I recovered the SPRB vertical position with step motors.

SUSP 

At the beginning of the shift, I contacted Paolo and Valerio regarding the issue with Sa_MC_F0_COIL_H1; to recover the correction, it would have been necessary to exit Science mode and perhaps unlock the ITF. We decided to continue monitoring the signal; around 16:00 UTC, the correction peaked and reversed the trend, so it was decided to postpone the intervention until tomorrow.

Guard tours (time in UTC)

13:00 - 13:30

17:00 - 17:30

20:00 - 20:30

21:40 - 22:10

ITF locked for all the shift.

At 8:27 UTC Diana from remote changed the TCSChiller set point to recover the standard working condition after the problem occurred yesterday evening (see entry 63940); it was decided to perform the operation without breaking Science mode.

Then, from 8:42 UTC to 8:45 UTC, Science mode was interrupted for an Adjusting break to adjust WI CO2 DAS outer ring power injected in the ITF; see plot.

Software

• TCSMoni, SqzMoni, DMSserver restarted many times to take into account new configurations provided by the experts;
• ELE_QGBT restarted after a crash at 8:26 UTC.

Following the activity of yesterday night (63940), we noticed that the WI main CO2 laser continued to be in an unstable state, so we decided to act on WI CO2 chiller setpoint, changing it from 19.02 °C to 19.04°C (08.27 UTC, Fig. 1). Also the BCK chiller set point has been changed at the same setpoint.

Soon after, we noticed a prominent change in the WI CO2 signals and we needed to recover the WI CO2 DAS outer ring power injected in the ITF, by rotating back the relative wave plate to recover the condition with respect to yesterday’s action (08.45 UTC, Fig. 2).

We’ll continue to monitor the situation to evaluate if a further chiller setpoint small change will be needed.

The locking troubles of the previous shifts were still in present at the beginning of this shift.

The commissioning team worked all the morning to find the source of the problem; right at the end of the shift we could lock in LN3_SQZ (see next dedicated entry for details) and set Science mode.

ISYS

At 7:48 UTC, taking advantage of an unlock, the INJ team recovered the BPC position.

Today we have concluded the tuning of DMS threshold for a few ENV channels which have been observed to fluctuate between green and red inside the DeadChannel monitor.

• ENV_EQB2_ACC_Y,  old value 1E-4.   ----> new value 1E-5
• ENV_EQB2_MIC,  old value 6.5E-3  ----> new value 1E-4
• ENV_IB_CT_FINGER_ACC_Y, old value 1.5E-3 ---> new value 1E-4

In particual for channel ENV_IB_CT_FINGER_ACC_Y we had to first fix the issue with too large 50 Hz and harmonics. The same problem was obseved in the past 62620 and fixed by replacing the BNC to lemo adaptor. Figure 1 shows the spectrum of the accelerometer powered off (blue) with the old connector, and Figure 2 shows the same with the new connector.  Figure 3 shows the procedure followed for setting threshold for the sensor ENV_IB_CT_FINGER_ACC_Y.

ITF found in Science mode.

At around 00:20 UTC Hrec_Range_BNS started to be very glitchy; the only indication was a red flag for the Squeezing thus at 1:25 UTC I set Adjusting mode and I tried to disengage the Squeezing. Unfurtunately this did not solve the problem: Hrec_Range_BNS continued to be noisy.

For this reason I removed the flag Science mode and I started to investigate the problem but I could not find the source.

At 2:40 UTC the ITF unlocked for a PR trip; the ITF relocked at the first attempt and the glitches on the Hrec_Range_BNS were no more there so at 3:24 UTC I set Science mode.

SUSP

The DMS is reporting: Sa_PR_F0_COIL_V2  Coil_close_to_saturation; experts informed by mail.

QNR

The DMS is reporting: SQZ_MIRROR1_X_is_drifting; experts informed by mail.

ITF found in calibration.

- At 21:12 UTC CALIBRATED_DF_INtoEND_LN concluded.

- At 21:24 UTC ITF in LN3_SQZ.

- At 21:25 UTC I launched the CALIBRATED_DF_SENSITIVITY but it failed; I tried to went back in LN3 disengaging the squeezing but the calibration failed again with following message:

2024-04-05T21:31:36.069Z CALI W: Traceback (most recent call last):

  File "/virgoApp/metatron/v0r3p0/Linux-x86_64-CL7/lib/python3.9/site-packages/guardian/worker.py", line 506, in run
    retval = statefunc()
  File "/virgoDev/Automation/userapps/CALI.py", line 1675, in main
    self.injection_scripts = ast.literal_eval(configUser.get('CALI', 'inj_sensitivity'))
  File "/virgo/conda/envs/automation/lib/python3.9/ast.py", line 62, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/virgo/conda/envs/automation/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "", line 16
    'python3 /virgoDev/Automation/scripts/CALI_O4/inject_checkHrecPcal.py          Inj  --duration  60 --config darkFringe --snr_pcal  25 --fmin 600  --fmax 1000  --semi_permanent_snr 100,
                                                                                                                                                                                            ^
SyntaxError: EOL while scanning string literal

experts informed by mail

- At 21:30 UTC CALIBRATED_DF_PCAL, concluded at 22:05;

- As the CALIBRATED_DF_SENSITIVITY failed at 22:07 UTC I launched the CALIBRATED_DF_DAILY; ITF unlocked at 22:20 UTC becasue of PR trip during the calibration (see attached plot1).

At the relock the ITF unlocked in twice ACQUIRE_LN3 during the misalignment os SR_TY; then it started to unlocks at varios steps: CARM_NULL_1F and also at step below for no obvious reasons (no seismic activity).

At 1:13 UTC I reached again LN2 and I waited a bit more before going in LN3 but then the ITF unlocked again in LN3 while misaligning the SR; at this point I contacted the ISC on-call.

Julia worked all the night but she could find any reasons of the unlocks; we decided to leave the ITF in LN1 to check if the problems might came from OMC, from the switch to LOW_NOISE, etc...

Deeper investigation will follow during the day.

ISC
(06-04-2024 01:30 - 04:30) From remote
Status: On-going
Description: locking troubles